Overview
TensorFlow is a software library for machine learning and artificial intelligence. It can be used across a range of tasks but is primarily used for training and inference of neural networks. It is one of the most popular deep learning frameworks, alongside others such as PyTorch. TensorFlow is free and open-source software released under the Apache License 2.0.
History
DistBelief
Starting in 2011, Google Brain built DistBelief as a proprietary machine learning system based on deep neural networks. Its use grew rapidly across diverse Alphabet companies in both research and commercial applications. Google assigned multiple computer scientists, including Jeff Dean, to simplify and refactor the codebase of DistBelief into a faster, more robust application-grade library, which became TensorFlow. In 2009, the team, led by Geoffrey Hinton, had implemented generalized backpropagation and other improvements, which allowed generation of neural networks with substantially higher accuracy, for instance a 25% reduction in errors in speech recognition.
TensorFlow
TensorFlow is Google Brain's second-generation system. Version 1.0.0 was released on February 11, 2017. While the reference implementation runs on single devices, TensorFlow can run on multiple CPUs and GPUs (with optional CUDA and SYCL extensions for general-purpose computing on graphics processing units). TensorFlow is available on 64-bit Linux, macOS, Windows, and mobile computing platforms including Android and iOS. Its flexible architecture allows for easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices. TensorFlow computations are expressed as stateful dataflow graphs. The name TensorFlow derives from the operations that such neural networks perform on multidimensional data arrays, which are referred to as tensors. During the Google I/O Conference in June 2016, Jeff Dean stated that 1,500 repositories on GitHub mentioned TensorFlow, of which only 5 were from Google. In March 2018, Google announced TensorFlow.js version 1.0 for machine learning in JavaScript. In January 2019, Google announced TensorFlow 2.0, which became officially available in September 2019. In May 2019, Google announced TensorFlow Graphics for deep learning in computer graphics.
Tensor processing unit (TPU)
In May 2016, Google announced its Tensor processing unit (TPU), an application-specific integrated circuit (ASIC) built specifically for machine learning and tailored for TensorFlow. A TPU is a programmable AI accelerator designed to provide high throughput of low-precision arithmetic (e.g., 8-bit), and oriented toward using or running models rather than training them. Google announced they had been running TPUs inside their data centers for more than a year, and had found them to deliver an order of magnitude better-optimized performance per watt for machine learning. In May 2017, Google announced the second-generation TPUs, as well as their availability in Google Compute Engine. The second-generation TPUs deliver up to 180 teraflops of performance, and when organized into clusters of 64 TPUs, provide up to 11.5 petaflops. In May 2018, Google announced the third-generation TPUs delivering up to 420 teraflops of performance and 128 GB high bandwidth memory (HBM). Cloud TPU v3 Pods offer 100+ petaflops of performance and 32 TB HBM. In February 2018, Google announced that they were making TPUs available in beta on the Google Cloud Platform.
Edge TPU
In July 2018, the Edge TPU was announced. Edge TPU is Google's purpose-built ASIC chip designed to run TensorFlow Lite machine learning models on small client computing devices such as smartphones, known as edge computing.
TensorFlow Lite
In May 2017, Google announced TensorFlow Lite as a software stack to support machine learning models for mobile and embedded devices, and in November 2017, provided the developer preview. In January 2019, the TensorFlow team released a developer preview of the mobile GPU inference engine with OpenGL ES 3.1 Compute Shaders on Android devices and Metal Compute Shaders on iOS devices. In May 2019, Google announced that their TensorFlow Lite Micro (also known as TensorFlow Lite for Microcontrollers) and ARM's uTensor would be merging. It was renamed as LiteRT in 2024.
TensorFlow 2.0
As TensorFlow's market share among research papers was declining to the advantage of PyTorch, the TensorFlow Team announced a release of a new major version of the library in September 2019. TensorFlow 2.0 introduced many changes, the most significant being TensorFlow eager, which changed the automatic differentiation scheme from the static computational graph to the "Define-by-Run" scheme originally made popular by Chainer and later PyTorch. Other major changes included removal of old libraries, cross-compatibility between trained models on different versions of TensorFlow, and significant improvements to the performance on GPU.
Features
AutoDifferentiation
AutoDifferentiation is the process of automatically calculating the gradient vector of a model with respect to each of its parameters. With this feature, TensorFlow can automatically compute the gradients for the parameters in a model, which is useful to algorithms such as backpropagation which require gradients to optimize performance. To do so, the framework must keep track of the order of operations done to the input Tensors in a model, and then compute the gradients with respect to the appropriate parameters.
Eager execution
TensorFlow includes an "eager execution" mode, which means that operations are evaluated immediately as opposed to being added to a computational graph which is executed later. Code executed eagerly can be examined step-by-step through a debugger, since data is augmented at each line of code rather than later in a computational graph. This execution paradigm is considered to be easier to debug because of its step-by-step transparency.
Distribute
In both eager and graph executions, TensorFlow provides an API for distributing computation across multiple devices with various distribution strategies. This distributed computing can often speed up the execution of training and evaluating of TensorFlow models and is a common practice in the field of AI.
Losses
To train and assess models, TensorFlow provides a set of loss functions (also known as cost functions). Some popular examples include mean squared error, categorical cross-entropy, and hinge loss. These functions measure the difference between predicted and actual values, guiding the optimization process during training.
Metrics
TensorFlow includes a metrics module that provides functions for evaluating model performance, such as accuracy, precision, recall, and area under the ROC curve. These metrics are commonly used to monitor training and validation progress.
Optimizers
TensorFlow offers a variety of optimizers to update model parameters during training, including stochastic gradient descent (SGD), Adam, RMSProp, and Adagrad. These optimizers implement different update rules to minimize the loss function efficiently.
Keras
TensorFlow integrates with Keras, a high-level neural network API that simplifies model building and training. Keras provides a user-friendly interface for defining layers, models, and training loops, making it accessible to both beginners and experts.
TensorBoard
TensorBoard is a visualization tool included with TensorFlow that allows users to monitor training metrics, visualize model graphs, and inspect embeddings. It helps in debugging and understanding model behavior.
TensorFlow Serving
TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. It supports model versioning, rolling out, and can handle multiple models simultaneously.
TensorFlow Hub
TensorFlow Hub is a library for reusable machine learning modules. It allows users to share and discover pre-trained model components, facilitating transfer learning and reducing the need to train models from scratch.
TensorFlow Extended (TFX)
TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines. It includes components for data validation, preprocessing, training, and serving, enabling scalable and reliable ML workflows.
Applications
TensorFlow is used in a wide range of applications, including image recognition, natural language processing, speech recognition, and recommendation systems. It is also employed in research and industry for tasks such as object detection, sentiment analysis, and generative modeling. Its flexibility and scalability make it suitable for both experimentation and large-scale deployment.
See also
- Artificial intelligence
- Machine learning
- Deep learning
- Neural network
- Large language model
- Transformer
- OpenAI
- Google DeepMind
- Generative AI
- PyTorch
References
- TensorFlow official documentation
- Wikipedia: TensorFlow
- Google AI Blog
- TensorFlow release notes