Spark Architecture & Execution model

Spark tutorial · PySpark.in

Spark Architecture & Execution model

Apache Spark works in a master-slave architecture where the master is called “Driver” and slaves are called “Workers".

Master manages, maintains, and monitors the slaves while slaves are the actual workers who perform the processing tasks. You tell the master what wants to be done and the master will take care of the rest. It will complete the task, using its slaves.

Apache Spark Architecture is based on two main abstractions :

Resilient Distributed Dataset (RDD)
Directed Acyclic Graph (DAG)

Resilient Distributed Dataset (RDD):

RDDs are the building blocks of any Spark application. RDDs Stands

for:

Resilient: Fault tolerant and is capable of rebuilding data on failure

Distributed: Distributed data among the multiple nodes in a cluster

Dataset: Collection of partitioned data with values

When you load the data into a Spark application, it creates an RDD which stores the loaded data.
RDD is immutable, meaning that it cannot be modified once created, but it can be transformed at any time.

Transformations :

They are the operations that are applied to create a new RDD. This function is used to transform the data into new RDD without altering the original data. e.g. groupBy, sum, rename, sort, etc

2) Actions :

They are applied on an RDD to instruct Apache Spark to apply computation and send the result back to the driver.

e.g. write, collect, show,etc

Types of transformations:

Narrow transformations –

Narrow transformations are those for which each input partition will contribute to only one output partition.
These transformations are performed on individual partition of data in an RDD in parallel.
Since they do not require shuffling of data between partitions, their performance is more efficient than wide transformation .
Narrow transformations can be performed on various data formats, such as unstructured, structured, and semi-structured data.

Wide transformations –

Wide transformation will have input partitions contributing to many output partitions.
These transformations require shuffling data between partitions.
They are typically more complex and require more resources to perform.
Wide transformations, however, typically require structured data in order to perform the necessary operations.

Lazy Evaluation:

Lazy Evaluation in Sparks means Spark will not start the execution of the process until an ACTION is called.

Spark is not too concerned as long as we are just performing transformations on the RDD, data frame or dataset.

Once Spark notices an ACTION being called, it starts looking at all the transformations, creates a DAG and execute it.

Directed Acyclic Graph (DAG) :

Directed Acyclic Graph is a finite direct graph that performs a sequence of computations on data.
The DAG in Spark supports cyclic data flow. Every Spark job creates a DAG of task stages that will be executed on the cluster.
Spark DAGs can contain many stages, unlike the Hadoop Map Reduce which has only two predefined stages.
In a Spark DAG, there are consecutive computation stages that optimize the execution plan.
You can achieve fault-tolerance in Spark with DAG.

Components of Spark Application Architecture :

Spark Application :

Spark application is a program built with Spark APIs and runs in a Spark compatible cluster/environment. It can be a PySpark script, a Java application, a Scala application, a Spark Session started by spark-shell or spark-sql command,etc.
It consists of a driver container and executors.

SparkSession / SparkContext :

An object that provides a point of entry to interact with underlying Spark functionality and allows programming Spark with its APIs.
It represents the connection to the Spark cluster. This class is how you communicate with some of Spark’s lower-level APIs, such as RDD

Job:

A parallel computation consisting of multiple tasks that gets produced in response to a Spark action (e.g., save(), collect()).

Stage:

Each job gets divided into smaller sets of tasks called stages that depend on each other.
Stages in Spark represent groups of tasks that can be executed together to compute the same operation on multiple machines.

Task:

A single unit of work or execution that runs in a Spark executor. Each stage contains one or multiple tasks.
Each task maps to a single core and works on a single partition of data

Driver :

The Driver is the main program that runs on the master node and is responsible for coordinating the entire Spark application. The Driver is responsible for several tasks, including :

Managing the SparkContext: The Driver is responsible for creating and managing the Spark Context, which is the main entry point fora Spark application.

Breaking down the application into tasks: The Driver is responsible for breaking down the application into a set of tasks that can be executed in parallel on the worker nodes.

Scheduling tasks: The Driver is responsible for scheduling tasks to worker nodes, based on the available resources and the requirements of tasks.

Monitoring tasks: The Driver is responsible for monitoring the tasks and making sure that they are executing correctly. If a task fails, the Driver can reschedule it on a different node.

Gathering results: The Driver is responsible for gathering results of the tasks and combining them to produce the final result of the application.

The Driver is the central component of a Spark application, and it playsa critical role in ensuring that the application runs correctly and efficiently.

Executor :

The Executor is a program that runs on the worker nodes and is responsible for executing the tasks assigned by the Driver. The Executor is responsible for several tasks, including:

Running tasks: The Executor is responsible for running the tasks assigned by the Driver. Each Executor runs a set of tasks, and it can run multiple tasks in parallel, based on the number of cores available on the node.

Reporting status: The Executor is responsible for reporting the status of the tasks to the Driver. The Driver uses this information to monitor the progress of the application and make sure that it is executing correctly.

Storing intermediate data: The Executor can store intermediate data in memory, which can be used by other tasks. This allows Spark to avoid shuffling data between nodes, which can be a performance bottleneck.

Releasing resources: The Executor is responsible for releasing resources when the tasks are completed, which allows Spark to make the most efficient use of the available resources.

The Executor is a critical component of a Spark application, and it plays a crucial role in executing the tasks and producing the final result.

Cluster Manager :

The cluster manager is responsible for managing the resources required for a Spark application, including CPU, memory, and network resources. Its primary functions include :

Resource allocation: The cluster manager receives resource requests from the Spark driver and allocates the necessary resources, such as CPU cores and memory, to the application.

Executor management: The cluster manager launches and manages Spark executors on worker nodes, which are responsible for executing tasks and storing data.

Fault tolerance: The cluster manager monitors the health of the worker nodes and detects failures, ensuring the smooth execution of the application by reallocating resources and restarting failed tasks.

Node management: The cluster manager keeps track of the worker nodes' status and manages their lifecycle, handling node registration, de-registration, and decommissioning.

Types of cluster managers:

Standalone– a simple cluster manager included with Spark that makes it easy to set up a cluster.

Apache Mesos– a general cluster manager that can also run Hadoop MapReduce and service applications.

Hadoop YARN– the resource manager in Hadoop 2.

Kubernetes– an open-source system for automating deployment, scaling, and management of containerized applications.

Execution of Spark Application :

When the Driver Program in the Apache Spark architecture executes, it calls the real program of an application and creates a SparkContext. SparkContext contains all of the basic functions. The Spark Driver includes several other components, including a DAG Scheduler, Task Scheduler, Backend Scheduler, and Block Manager ,all of which are responsible for translating user-written code into jobs that are actually executed on the cluster.
The Cluster Manager manages the execution of various jobs in the cluster. Spark Driver works in conjunction with the Cluster Manager to control the execution of various other jobs. The cluster Manager does the task of allocating resources for the job. Once the job has been broken down into smaller jobs, which are then distributed to worker nodes, Spark Driver will control the execution.

Whenever an RDD is created in the SparkContext, it can be distributed across many worker nodes and can also be cached there. Worker nodes execute the tasks assigned by the Cluster Manager and return it back to the Spark Context.
The executor is responsible for the execution of these tasks. The lifespan of executors is the same as that of the Spark Application. We can increase the number of workers if we want to improve the performance of the system. In this way, we can divide jobs into more coherent parts.

Spark Execution Modes :

There are 3 types of execution modes –

Local Mode
Client Mode
Cluster Mod

1) Local Mode :

This is similar to executing a program on a single JVM on someone’s laptop or desktop.
It could be a program written in any language, such as Java, Scala or Python.
However, you should have defined and used a spark context object in these apps, as well as imported spark libraries and processed data from your local system files.
This is the local mode because everything is done locally, there is no concept of a node, and nothing is done in a distributed manner.
A single JVM process is used to produce both the driver and the executor.
For example, launching a spark-shell on your laptop is an example of a local mode of execution.

Client Mode:

In client mode, the driver is present on the client machine (laptop/desktop). i.e., the driver is not part of the cluster. On the other hand, the executors run within the cluster.
The driver connects to the cluster manager, starts all the executors on the clusters for interactive queries and receives the results back to the client.
In case of a problem with the client (local) machine or you log off ,the driver will go off and subsequently all executors will shut down on the cluster.

Cluster Mode:

In cluster mode, the driver and executor both run inside the cluster.
The spark job is submitted from your local machine to a cluster machine within the cluster. Such machines are usually called edge node .
In case of a problem with the client (local) machine or you log off ,the driver will not get impacted as it is running on the cluster.

This means that the cluster manager is responsible for maintaining all Spark Application related processes.
This mode is useful for long running queries and production environments.

Spark Architecture & Execution model

Spark Architecture & Execution model

More Spark tutorials