# Apache Spark

Apache Spark is an open-source unified analytics engine for large-scale data processing, providing an interface for programming clusters with implicit data parallelism and fault tolerance.

Apache Spark is an open-source unified analytics engine for large-scale data processing. It provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab starting in 2009, the Spark codebase was donated to the Apache Software Foundation in 2013, which has maintained it since.

Spark's architectural foundation is the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, maintained in a fault-tolerant way. The DataFrame API was released as an abstraction on top of the RDD, followed by the Dataset API. In Spark 1.x, the RDD was the primary API, but as of Spark 2.x, use of the Dataset API is encouraged, though the RDD API is not deprecated. The RDD technology still underlies the Dataset API.

Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a linear dataflow structure on distributed programs. MapReduce programs read input data from disk, map a function across the data, reduce the results, and store reduction results on disk. Spark's RDDs function as a working set for distributed programs, offering a deliberately restricted form of distributed shared memory.

Inside Spark, the workflow is managed as a directed acyclic graph (DAG). Nodes represent RDDs, while edges represent operations on the RDDs. Spark facilitates both iterative algorithms, which visit their data set multiple times in a loop, and interactive or exploratory data analysis, such as repeated database-style querying. The latency of such applications may be reduced by several orders of magnitude compared to Apache Hadoop MapReduce implementations. Among iterative algorithms are training algorithms for [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) systems, which formed the initial impetus for developing Spark.

## Spark Core

Spark Core is the foundation of the overall project. It provides distributed task dispatching, scheduling, and basic I/O functionalities, exposed through an application programming interface for Java, Python, Scala, .NET, and R, centered on the RDD abstraction. The Java API is available for other JVM languages and some non-JVM languages that can connect to the JVM, such as Julia. This interface mirrors a functional or higher-order model of programming: a driver program invokes parallel operations such as map, filter, or reduce on an RDD by passing a function to Spark, which schedules the function's execution in parallel on the cluster. These operations, and additional ones such as joins, take RDDs as input and produce new RDDs. RDDs are immutable and their operations are lazy; fault tolerance is achieved by keeping track of the lineage of each RDD (the sequence of operations that produced it) so it can be reconstructed in case of data loss. RDDs can contain any type of Python, .NET, Java, or Scala objects.

Besides the RDD-oriented functional style, Spark provides two restricted forms of shared variables: broadcast variables reference read-only data needed on all nodes, while accumulators can be used to program reductions in an imperative style.

A typical example of RDD-centric functional programming is a Scala program that computes word frequencies from text files and prints the most common ones. Each map, flatMap, and reduceByKey takes an anonymous function that performs a simple operation on a single data item (or a pair of items) and applies it to transform an RDD into a new RDD.

## PySpark

PySpark is the Python API for Apache Spark, introduced in Spark 0.7, released in February 2013. It allows Spark applications to be written in Python and provides an interactive shell for exploratory analysis. PySpark code executes under the standard CPython runtime and can call native Python libraries such as NumPy and SciPy; communication between Python user code and Spark's JVM-based core has historically been mediated by the Py4J library.

PySpark provides Python interfaces to most of Spark's components, including Spark Core, Spark SQL and the DataFrame API, Structured Streaming, and MLlib; since Spark 4.1 it also exposes the Spark Declarative Pipelines API. A pandas-compatible API, known as pandas API on Spark, was integrated into PySpark in Apache Spark 3.2.0, released in October 2021. It provides equivalents to the pandas data-analysis API that execute against the Spark engine, allowing pandas-style code to be run on distributed data. The component originated as Koalas, a separate project open-sourced by Databricks in 2019, which was subsequently merged into PySpark. PySpark also provides a client for Spark Connect, a client-server architecture enabling remote connectivity to Spark clusters from PySpark applications.

A typical PySpark program uses a SparkSession to load data as a DataFrame and apply transformations.

## Spark SQL

Spark SQL is a component on top of Spark Core that introduced a data abstraction called DataFrames, which provides support for structured and semi-structured data. Spark SQL provides a domain-specific language (DSL) to manipulate DataFrames in Scala, Java, Python, or .NET. It also provides SQL language support, with command-line interfaces and ODBC/JDBC server. Although DataFrames lack the compile-time type-checking afforded by RDDs, as of Spark 2.0, the strongly typed Dataset is fully supported by Spark SQL as well.

## Spark Declarative Pipelines

Spark Declarative Pipelines (SDP) is a declarative framework for defining and running extract, transform, load (ETL) data pipelines, added in Apache Spark 4.1.0, which was released on December 16, 2025. Developers declare the datasets a pipeline processes and the transformations to apply, allowing the engine to optimize execution. This framework aims to simplify pipeline development and maintenance by reducing boilerplate code and enabling better optimization opportunities.

## Cluster Management and Storage

Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone native Spark, Hadoop YARN, Apache Mesos, or Kubernetes. A standalone native Spark cluster can be launched manually or by the launch scripts provided by the install package. It is also possible to run the daemons on a single machine for testing. For distributed storage, Spark can interface with a wide variety of distributed systems, including Alluxio, Hadoop Distributed File System (HDFS), MapR File System (MapR-FS), Cassandra, OpenStack Swift, Amazon S3, Kudu, Lustre file system, or a custom solution. Spark also supports a pseudo-distributed local mode, usually used only for development or testing, where distributed storage is not required and the local file system can be used; in such a scenario, Spark runs on a single machine with one executor per CPU core.

Spark's ability to handle both batch and streaming workloads, combined with its in-memory processing and rich APIs, has made it a popular choice for large-scale data processing in industries ranging from finance to [cloud computing](https://www.wikiprompt.org/wiki/amazon-web-services). Its integration with [apache-kafka](https://www.wikiprompt.org/wiki/apache-kafka) for streaming data and [apache-airflow](https://www.wikiprompt.org/wiki/apache-airflow) for workflow orchestration further extends its utility in modern data pipelines.

## External links

- [Wikipedia: Apache Spark](https://en.wikipedia.org/wiki/Apache_Spark)

---
Source: https://www.wikiprompt.org/wiki/apache-spark
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-10-13T12:03:29.451026+00:00