# Dataflow (SambaNova)

Dataflow is SambaNova Systems' reconfigurable dataflow architecture for AI, designed to accelerate machine learning workloads by mapping computation graphs onto a grid of processing elements with on-chip memory, reducing data movement and improving efficiency.

Dataflow is a computing architecture developed by [SambaNova Systems](https://www.wikiprompt.org/wiki/samba-nova) for accelerating [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) workloads. Unlike conventional processor designs that execute instructions sequentially, Dataflow organizes computation as a directed graph where data flows continuously through a network of processing elements. This approach is tailored for the structured, repetitive operations found in [deep-learning](https://www.wikiprompt.org/wiki/deep-learning) models, such as [neural networks](https://www.wikiprompt.org/wiki/neural-network) and [large language models](https://www.wikiprompt.org/wiki/large-language-model). The architecture is implemented in SambaNova's custom hardware, including the DataScale systems and the SN10 series of reconfigurable dataflow units (RDUs).

The Dataflow architecture emerged from research at [Stanford University](https://www.wikiprompt.org/wiki/stanford-ai-lab) and was commercialized by SambaNova, which was founded in 2017 by Kunle Olukotun, Christopher Ré, and Rodrigo Liang. The company's first product, the DataScale SN10, was announced in 2020 and began shipping in 2021. The architecture is designed to address the memory bandwidth bottleneck that limits traditional GPU-based systems, which often spend significant energy moving data between memory and compute units. By keeping data on-chip and streaming it through a configurable fabric, Dataflow aims to achieve higher utilization and lower latency for AI inference and training.

## Design Principles

The core principle of Dataflow is to eliminate the von Neumann bottleneck by decoupling computation from instruction fetch and decode. In a conventional CPU or GPU, each operation requires fetching an instruction, decoding it, and then executing it on data fetched from memory. Dataflow instead compiles the entire model into a static dataflow graph, where each node represents a mathematical operation (such as matrix multiplication or activation) and each edge represents a data dependency. The graph is then mapped onto a grid of processing elements (PEs), each with its own local memory and arithmetic units.

Each PE in the Dataflow architecture can perform a variety of operations, including multiply-accumulate, activation functions, and pooling. The PEs are interconnected via a high-bandwidth, low-latency network that supports both nearest-neighbor and longer-range communication. The configuration of this network is determined at compile time, allowing the hardware to be specialized for each specific model. This reconfigurability is a key differentiator from fixed-function accelerators like [AWS Trainium](https://www.wikiprompt.org/wiki/aws-trainium) or [Groq](https://www.wikiprompt.org/wiki/groq)'s tensor streaming processor, which also use dataflow principles but with different trade-offs.

## Compilation and Software Stack

SambaNova provides a software stack called SambaFlow, which compiles models written in frameworks like PyTorch and TensorFlow into Dataflow graphs. The compiler performs several optimizations, including operator fusion, memory layout optimization, and pipelining. It also handles the mapping of the graph onto the physical PE grid, balancing compute and memory resources. SambaFlow supports both training and inference, and it includes a runtime that manages data movement and synchronization.

The compilation process is static, meaning that the graph is fully determined before execution begins. This allows the compiler to schedule operations precisely and avoid runtime overhead. However, it also means that dynamic control flow, such as loops with variable trip counts, must be unrolled or handled via special mechanisms. SambaNova has extended its compiler to support transformer models, which are the basis of most modern [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) systems, by optimizing attention mechanisms and feed-forward layers.

## Hardware Implementation

The SN10 RDU, introduced in 2020, contains 40 billion transistors and is fabricated using a 7-nanometer process at [TSMC](https://www.wikiprompt.org/wiki/tsmc). It features 48 GB of on-chip SRAM, which is distributed across the PE grid. The chip operates at a clock speed of around 1.2 GHz, but its performance is measured in terms of dataflow throughput rather than raw clock rate. SambaNova claims that a single SN10 can deliver up to 638 teraflops of compute for sparse models and 125 teraflops for dense models, though these figures depend on the workload.

The DataScale system integrates multiple SN10 chips in a single chassis, with up to 16 RDUs per system. These systems are connected via a high-speed interconnect that supports peer-to-peer communication. Later generations, such as the SN30 and SN40, have increased memory and compute capacity, with the SN40 introduced in 2023 featuring 1.5 TB of on-chip memory and support for models with up to 5 trillion parameters. The hardware also includes specialized units for [pruning](https://www.wikiprompt.org/wiki/model-pruning) and sparsity, which can skip zero-valued computations to improve efficiency.

## Performance Characteristics

Dataflow systems are particularly effective for inference workloads, where the same model is executed repeatedly with different inputs. The static scheduling allows for predictable latency, and the on-chip memory reduces the need to access external DRAM. In benchmarks published by SambaNova, the DataScale SN10 achieves higher throughput than NVIDIA A100 GPUs for several popular models, including BERT and GPT-3, while consuming less power. However, independent benchmarks have been limited, and the actual performance can vary depending on the model and batch size.

For training, Dataflow faces challenges due to the need for frequent weight updates and the dynamic nature of gradient computation. SambaNova has addressed this by supporting data parallelism and model parallelism, where multiple RDUs work on different parts of the model. The compiler can pipeline the forward and backward passes to overlap computation and communication. Despite these efforts, training performance has been less competitive than inference, and many customers use Dataflow primarily for deployment.

## Comparison with Other Architectures

Dataflow is often compared with [Graphcore](https://www.wikiprompt.org/wiki/graphcore)'s IPU, which also uses a many-core design with on-chip memory. The key difference is that Graphcore uses a more traditional instruction-based execution model, while Dataflow is purely data-driven. Groq's tensor streaming processor is another dataflow-like architecture, but it uses a simpler, more rigid design with no on-chip memory, relying instead on streaming data through a large array of multiply-accumulate units. SambaNova's reconfigurability offers more flexibility for different model shapes, but it also adds complexity to the compiler.

Compared to GPUs from [NVIDIA](https://www.wikiprompt.org/wiki/nvidia) (not in the provided list, but implied) or [AMD](https://www.wikiprompt.org/wiki/amd), Dataflow offers lower latency for small batches and better energy efficiency for inference. However, GPUs benefit from a mature software ecosystem and broader support for frameworks and libraries. Dataflow's advantage is most pronounced for large transformer models, where the attention mechanism can be mapped efficiently onto the PE grid.

## Applications and Adoption

SambaNova has deployed Dataflow systems in various sectors, including government, healthcare, and financial services. Notable customers include the U.S. Department of Energy, which uses the system for scientific computing, and [Oracle Cloud](https://www.wikiprompt.org/wiki/oracle-cloud), which offers SambaNova hardware as a managed service. The architecture has been used for tasks such as natural language processing, computer vision, and drug discovery. In 2023, SambaNova introduced Samba-1, a large language model with 1 trillion parameters, trained on its Dataflow systems, demonstrating the architecture's capability for large-scale training.

The company has also focused on making Dataflow accessible through cloud services, allowing customers to rent time on the hardware without purchasing it outright. This model has been adopted by enterprises that need high-performance AI inference but lack the resources to maintain dedicated infrastructure. As of 2024, SambaNova has raised over $1 billion in funding and continues to iterate on its hardware and software.

## Challenges and Limitations

One of the main challenges for Dataflow is the complexity of the compiler. Mapping arbitrary models onto the PE grid requires sophisticated algorithms for partitioning, scheduling, and memory allocation. This complexity can lead to longer compilation times, especially for large models. Additionally, the static nature of the graph makes it difficult to support models with dynamic shapes or control flow, which are common in some applications like reinforcement learning.

Another limitation is the lack of ecosystem support compared to GPUs. Many AI researchers and developers are familiar with CUDA and PyTorch, and they may be reluctant to adopt a new architecture that requires a different programming model. SambaNova has mitigated this by providing compatibility layers, but there are still gaps. The company has also faced competition from more established players like [Google Cloud](https://www.wikiprompt.org/wiki/google-cloud) and [AWS](https://www.wikiprompt.org/wiki/amazon-web-services), which offer their own custom accelerators.

## Future Directions

SambaNova continues to evolve the Dataflow architecture, with a focus on improving training efficiency and supporting larger models. The SN40 generation includes enhancements for sparse computation and mixed-precision arithmetic. The company is also exploring the use of Dataflow for edge devices, though the current hardware is primarily designed for data centers. As [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) models grow in size, the demand for efficient inference hardware is likely to increase, and Dataflow's approach may become more attractive.

Research into dataflow architectures is also ongoing in academia, with groups at [MIT CSAIL](https://www.wikiprompt.org/wiki/mit-csail) and [Berkeley AI Research](https://www.wikiprompt.org/wiki/berkeley-ai-research) exploring similar ideas. The principles of dataflow computing have a long history, dating back to the 1970s, but it is only recently that advances in chip manufacturing and compiler technology have made them practical for commercial AI workloads. SambaNova's Dataflow represents one of the most prominent commercial implementations of this concept, and its success will depend on its ability to adapt to the rapidly changing AI landscape.

## Conclusion

Dataflow is a distinctive approach to AI acceleration that prioritizes data movement efficiency and reconfigurability. By mapping models onto a grid of processing elements with on-chip memory, it offers a compelling alternative to traditional GPUs for inference tasks. While it faces challenges in compiler complexity and ecosystem adoption, its performance advantages for large language models have attracted significant investment and customer interest. As AI models continue to grow, the principles underlying Dataflow may become increasingly relevant to the broader computing industry.

---
Source: https://www.wikiprompt.org/wiki/samba-nova-dataflow
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T01:55:00.158525+00:00