Apache Spark Runtime Architecture

Spark tutorial · PySpark.in

What is Apache Spark?

Apache Spark is a distributed computing framework.

That means:

• Every Spark application is a distributed application.

• Distributed applications need a cluster to run efficiently.

 You can run Spark on your local machine (for testing), but in production, Spark applications   run  on clusters.

What is a Cluster?

A cluster is a group of networked machines (called worker nodes) that work together.

 Example Cluster:

• 10 machines (workers)

• Each machine: 16 CPU cores + 64 GB RAM

• Total capacity = 160 CPU cores + 640 GB RAM

• A diagram of 10 servers connected together, showing total CPU & RAM capacity.
  
  

Cluster Managers in Spark

A Cluster Manager handles resource allocation (CPU + memory).

Most commonly used:

1. Hadoop YARN

2. Kubernetes

Others: Apache Mesos, Spark Standalone
(But YARN + Kubernetes cover 90%+ use cases).

Submitting a Spark Application

 Step 1: Request Sent to Cluster Manager

• The spark-submit command sends a request to the Cluster Resource Manager (e.g., YARN, Kubernetes, or Mesos).

• The cluster manager is responsible for allocating resources (CPU, memory) for your Spark job.

 Step 2: Application Master (AM) Created

• On YARN, the cluster manager launches an Application Master (AM) container on one of the worker nodes.

• The AM is like the “coordinator” for your Spark job. It manages resources and monitors the execution of your application.

 Step 3: Driver Program Starts

• Inside the AM container, your Driver Program starts running.

• The Driver:

  • Translates your PySpark/Scala code into tasks.

  • Requests executors (worker processes) from YARN.

  • Sends tasks to those executors.

Step 4: Executors Run Tasks

• YARN allocates Executor Containers on worker nodes.

• Executors actually perform the computations (running transformations and actions on RDDs/DataFrames).

• They also report back the results to the Driver.

Step 5: Job Completion

• Once all tasks are completed:

  • Results are collected back by the Driver.

  • Application Master tells YARN the job is finished.

  • Executors and AM containers are released.

Simplified Flow

1. spark-submit → Sends job to Cluster Manager

2. Cluster Manager (YARN) → Starts Application Master

3. Application Master → Launches Driver Program

4. Driver Program → Requests Executors & distributes tasks

5. Executors → Run tasks and return results

6. Application Ends → Resources released

Note:

• spark-submit → YARN → Application Master → Driver → Executors → Results

More Spark tutorials

• about pyspark
• text diagram
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark
• DataFrame Operations in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges