PySpark & Apache Spark
Interview Questions

The easiest way to understand Spark architecture is:

There are three important components:

1. Driver
2. Cluster Manager
3. Executors

1️⃣ Driver

The Driver is the brain of the Spark application.

Suppose you write:

df = spark.read.parquet("s3://data/sales")

result = (
    df.filter("amount > 1000")
      .groupBy("country")
      .sum("amount")
)

result.show()

The Driver:

• Runs your main application code
• Creates the SparkSession
• Builds the logical/physical execution plan
• Creates jobs
• Divides jobs into stages
• Coordinates tasks
• Communicates with executors

Think:

Driver = Brain

2️⃣ Cluster Manager

The cluster manager is responsible for providing/allocating cluster resources.

Examples:

YARN
Kubernetes
Standalone

The Driver asks the cluster manager for resources.

Conceptually:

Driver
  │
  │ "I need resources"
  ↓
Cluster Manager
  │
  ├── Executor 1
  ├── Executor 2
  └── Executor 3

3️⃣ Executors

Executors are processes running on worker nodes that actually execute the tasks.

For example:

Executor 1
 ├── Task 1
 ├── Task 2
 ├── Task 3
 └── Task 4

Executors also provide memory for things such as:

• Cached/persisted data
• Shuffle data
• Intermediate computation
• Execution memory

Think:

Executor = Worker

Simple analogy

Imagine a restaurant.

Driver          = Manager
Cluster Manager = Resource/Staff allocator
Executors       = Chefs
Tasks           = Individual cooking jobs

The manager doesn't cook every dish.

The manager coordinates the work.

Similarly:

Driver coordinates; executors execute.

Interview answer

If interviewer asks:

"Explain Spark architecture."

Say:

"A Spark application consists primarily of a Driver, executors, and a cluster manager. The Driver runs the application's main code, builds the execution plan, and coordinates jobs, stages, and tasks. The cluster manager allocates resources for the application. Executors run on worker nodes and execute the tasks assigned by the Driver while also providing memory for cached data and intermediate processing."

DAG stands for:

Directed Acyclic Graph

Don't worry about the name initially.

Think of it as:

A graph showing the sequence of transformations Spark needs to perform.

Suppose you write:

df.filter("amount > 1000") \
  .select("customer_id", "amount") \
  .groupBy("customer_id") \
  .sum("amount")

Spark doesn't immediately execute every line.

Because of lazy evaluation, Spark builds a plan.

Conceptually:

Read Data
    ↓
Filter
    ↓
Select
    ↓
GroupBy
    ↓
Aggregation

This is part of Spark's execution planning.

What does the DAG Scheduler do?

The DAG Scheduler analyzes the computation and divides it into stages based primarily on dependencies and shuffle boundaries.

For example:

Filter
  ↓
Select
  ↓
Shuffle
  ↓
GroupBy
  ↓
Aggregation

The scheduler can divide it approximately into:

Stage 1
Filter
  ↓
Select
  ↓
Shuffle

and:

Stage 2
GroupBy
  ↓
Aggregation

The resulting stage tasks are then submitted for execution through Spark's scheduling mechanisms

This is one of the most important Spark interview concepts.

Remember:

Application
     ↓
   Job
     ↓
  Stage
     ↓
  Task

Let's understand each.

Application

A Spark application is the entire Spark program.

For example:

spark = SparkSession.builder.getOrCreate()

df = spark.read.parquet("sales")

result = df.groupBy("country").sum("amount")

result.write.parquet("output")

This whole program is the Spark application.

Typically, one application has one SparkSession.

Job

A job is triggered by an action.

Examples of actions:

df.count()
df.collect()
df.show()
df.write.parquet(...)

For example:

df.count()

can trigger a Spark job.

So:

Transformation
Transformation
Transformation
     ↓
   Action
     ↓
   JOB

Important

Transformations such as:

filter()
select()
groupBy()
withColumn()

are lazy and generally don't execute immediately.

An action triggers execution.

Stage

A job is divided into stages.

The major boundary between stages is generally a shuffle.

For example:

df.filter(...)
  .groupBy("country")
  .sum("amount")

The groupBy may require data to move between partitions.

Conceptually:

Stage 1
filter
  ↓
shuffle
  ↓
Stage 2
groupBy + aggregation

So remember:

Shuffle → Stage boundary

Task

A stage contains multiple tasks.

Generally:

One task processes one partition for that stage.

Suppose a stage has:

4 partitions

then you generally get:

Task 1 → Partition 1
Task 2 → Partition 2
Task 3 → Partition 3
Task 4 → Partition 4

These tasks execute on executors.

Complete picture

 Interview answer

"A Spark application is the complete Spark program. An action triggers a job. A job is divided into stages, generally at shuffle boundaries. Each stage contains tasks, and typically one task processes one partition. Tasks are the units of work that execute on Spark executors."

The image mentions:

Let's understand each.

Spark Core

This is the foundation of Spark.

It handles things such as:

• Task scheduling
• Memory management
• Fault recovery
• RDDs
• Communication with executors

Think:

           Spark Core
               │
       ┌───────┼───────┐
       ↓       ↓       ↓
 Scheduling  Memory  Fault Recovery

Spark SQL

Used for:

• DataFrames
• SQL queries
• Structured data
• Query optimization

Example:

df = spark.read.parquet("sales.parquet")

df.groupBy("country").sum("amount")

Or:

SELECT country, SUM(amount)
FROM sales
GROUP BY country;

Spark SQL uses the Catalyst optimizer to optimize SQL/DataFrame query plans.

Structured Streaming

Used for processing continuously arriving data.

For example:

Kafka
  ↓
Spark Structured Streaming
  ↓
Transform
  ↓
Delta Lake

Typical use cases:

• Real-time transaction processing
• Log processing
• IoT
• Event processing

MLlib

Spark's machine-learning library.

Used for algorithms such as:

• Classification
• Regression
• Clustering
• Recommendation-related workflows

GraphX

Spark's graph-processing library.

Used for graph-oriented problems such as:

Customer → Account → Transaction

or network relationships.

For modern interviews, Spark SQL and Structured Streaming are generally much more relevant for data-engineering roles than spending a lot of time on GraphX.

 Interview answer

"The Spark ecosystem includes Spark Core as the execution foundation, Spark SQL for structured data and DataFrame/SQL processing, Structured Streaming for streaming workloads, MLlib for machine learning, and GraphX for graph processing."

This is another very common question.

SparkContext

SparkContext is the older/core entry point into Spark.

It provides access to things such as:

• Cluster resources
• RDD APIs
• Spark configuration
• Executors

Conceptually:

SparkContext
     │
     ↓
Spark Cluster
     │
     ├── Executor 1
     ├── Executor 2
     └── Executor 3

SparkSession

Modern Spark applications normally use:

spark = SparkSession.builder.getOrCreate()

SparkSession is the primary entry point for:

• DataFrames
• SQL
• Structured Streaming
• Spark configuration
• Catalog operations

It also provides access to the underlying SparkContext:

spark.sparkContext

Easy way to remember

SparkContext

"Core connection to the Spark cluster."

SparkSession

"Main entry point for modern Spark applications."

Relationship:

            SparkSession
                 │
       ┌─────────┼─────────┐
       ↓         ↓         ↓
   DataFrame    SQL     Streaming
                 │
                 ↓
          SparkContext
                 │
                 ↓
           Spark Cluster

 Interview answer

"SparkContext is the older core entry point that represents the connection to the Spark execution environment and is mainly associated with the RDD API. SparkSession is the modern unified entry point for DataFrame, SQL, and Structured Streaming applications. In modern PySpark applications, I normally use SparkSession and access SparkContext through spark.sparkContext when needed."

What is PySpark?

PySpark is the Python API for Apache Spark.

Spark itself is primarily implemented on the JVM, particularly in Scala and Java.

When you write:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read.parquet("s3://bucket/data/")

you are writing Python code, but Spark's execution engine operates on the JVM.

How does Python communicate with Spark?

Conceptually:

PySpark uses a gateway mechanism to communicate between Python and the JVM.

Does every DataFrame operation execute Python code row-by-row?

No.

This is extremely important.

Suppose you write:

df.select("customer_id", "amount")

Spark doesn't normally execute:

Python
 ↓
row 1
 ↓
Python
 ↓
row 2
 ↓
Python
 ↓
row 3

Instead, Spark builds a logical plan and executes the supported DataFrame operations through the Spark engine.

That's one reason DataFrame APIs are much more efficient than Python row-by-row processing.

What happens with a Python UDF?

Now consider:

from pyspark.sql.functions import udf

If you use a regular Python UDF, Python code needs to participate in processing the data.

Conceptually:

Spark JVM
   ↓
Python Worker
   ↓
Python function
   ↓
Result
   ↓
Spark JVM

That communication can introduce overhead.

That's why native Spark functions are generally preferred when possible.

For example, prefer:

from pyspark.sql import functions as F

df = df.withColumn(
    "upper_name",
    F.upper("name")
)

over a Python UDF for something Spark already provides natively.

🎯 Interview answer

"PySpark is the Python API for Apache Spark. Spark's core execution engine runs on the JVM. PySpark communicates with the JVM, and DataFrame operations are converted into Spark execution plans that run in the Spark engine. Regular Python UDFs can introduce additional Python-JVM communication and serialization overhead, so native Spark functions are generally preferred when available."

Simple explanation

Apache Spark is a distributed data-processing engine used to process large amounts of data across multiple machines.

For example, suppose you have 1 TB of transaction data.

Instead of processing it on one machine:

1 TB Data
   ↓
One machine
   ↓
Very slow

Spark distributes the data:

                 1 TB Data
                     ↓
              Spark Cluster
                     │
        ┌────────────┼────────────┐
        ↓            ↓            ↓
    Executor 1   Executor 2   Executor 3
       330 GB       330 GB       340 GB

The executors process their portions in parallel.

Why is Spark generally faster than traditional MapReduce?

The most important reason is that Spark can keep intermediate data in memory and avoid unnecessary disk I/O.

Traditional MapReduce

Conceptually:

Input
  ↓
Map
  ↓
Write intermediate data to disk
  ↓
Shuffle
  ↓
Write/read intermediate data
  ↓
Reduce
  ↓
Output

There can be substantial disk I/O between stages.

Spark

Spark builds a computation plan and can keep intermediate data in memory when appropriate:

Input
  ↓
Transformation
  ↓
Transformation
  ↓
Transformation
  ↓
Action

Spark also uses:

• DAG execution
• Lazy evaluation
• Query optimization
• In-memory caching when beneficial
• Efficient shuffle mechanisms

Important interview correction

Don't say:

❌ "Spark is always 100 times faster than MapReduce."

That's too absolute.

Say:

"Spark can be significantly faster than traditional MapReduce for many workloads, especially iterative and multi-stage workloads, because it can avoid repeated disk I/O and optimize the execution plan. The actual improvement depends on the workload, data size, cluster configuration, and implementation."

Interview answer

"Apache Spark is a distributed processing engine designed for large-scale data processing. Compared with traditional MapReduce, Spark can be faster because it builds a DAG of operations, optimizes execution, and can keep intermediate data in memory rather than repeatedly writing intermediate results to disk. This is particularly beneficial for iterative and multi-stage workloads."

Transformation

Operations that create a new RDD/DataFrame from an existing one. They are lazy, i.e., not executed immediately. They build a logical plan.

Action

Operations that trigger the execution of the DAG and return a result to the driver program or write data to storage. They are eager (executed immediately).

Transformation vs Action

Transformation Examples

df.filter(df.age > 18)

df.select("name", "age")

df.withColumn("salary", df.salary * 1.1)

df.join(other_df, "id")

df.groupBy("dept").count()

Action Examples

They are testing two things:

1. Performance mindset — Do you understand why Spark jobs become slow?
2. Troubleshooting approach — Can you find the actual bottleneck instead of blindly applying optimizations?

You can answer:

“First, I would check the Spark UI to identify which stage or task is taking the most time. I would look for issues such as large shuffles, data skew, excessive task execution time, spills, or too many partitions.

Then, based on the bottleneck, I would optimize the job. I would filter the data as early as possible, select only the required columns, reduce unnecessary shuffles, and use appropriate partitioning.

For joins, if one dataset is small enough, I would consider a broadcast join. If there is data skew, I would use an appropriate technique to handle the skew.

I would also avoid unnecessary actions and repeated computations, and use caching only when the same DataFrame is reused. For Delta tables, I would also look at the table layout and small-file problem.

Finally, I would run the job again and compare the execution time and resource usage before and after the optimization.”

With a broadcast join, Spark collects the smaller table on the driver, creates a broadcast variable, and distributes it to every executor. This avoids shuffling the smaller table across the cluster and can significantly improve join performance. However, the broadcasted table consumes memory on both the driver (during broadcast creation) and on every executor (where a copy is stored). If the broadcast table is too large, it can increase network traffic, memory usage, and garbage collection overhead, potentially leading to executor or driver out-of-memory errors and failed jobs. Therefore, before increasing spark.sql.autoBroadcastJoinThreshold, it's important to evaluate the size of the table being broadcast as well as the cluster's driver memory, executor memory, and overall resource configuration.