Lazy Evaluation in PySpark

Spark tutorial · PySpark.in

Lazy Evaluation in PySpark

When you build pipelines in PySpark, you often chain together several transformations on a DataFrame—filtering rows, selecting columns, joining tables, grouping values and more. Unlike many Python libraries, Spark does not execute these operations right away. Instead, it records them and waits until you explicitly ask for the result. This design is known as lazy evaluation and is key to Spark’s performance and scalability.

What is Lazy Evaluation?

Lazy evaluation means that transformations on an RDD, DataFrame or Dataset build a plan rather than compute results immediately. According to Spark documentation and tutorials:

Transformations are lazy by nature; they do not execute until an action is called. Instead, Spark keeps a record of each transformation in a lineage or logical plan.
Spark maintains these operations in a Directed Acyclic Graph (DAG) representing the dependencies between steps. Only when an action (such as count, show, collect) is invoked does Spark traverse this DAG and start the computation.
Because the transformations are just plans, no data is loaded or processed until it is needed. This allows Spark to combine or reorder operations for efficiency

In essence, you are writing a recipe of what you want to do, but Spark only cooks when you call an action.

How Lazy Evaluation Works

Record transformations: Each time you apply a transformation like filter() or groupBy(), Spark adds a node to the DAG. The data itself remains untouched; Spark simply notes what should happen.
Optimise the plan: Because nothing has executed yet, Spark can analyse the DAG, combine adjacent operations, and remove unnecessary steps. This optimisation reduces the number of passes over the data
Trigger execution with an action: The first action—such as show() or count()—triggers Spark to traverse the DAG from the beginning, launch tasks across the cluster and compute the result. Spark only reads the data partitions required for that result; for example, if you call first(), Spark may read only the first partition and ignore the rest

Why Lazy Evaluation is Beneficial

Lazy evaluation offers several practical advantages:

Fewer data scans: By combining multiple transformations before executing, Spark reduces the number of times it reads the underlying data=
Reduced computation: Only necessary computations are performed. If you load a 1 GB file and ask for the first line, Spark reads just the relevant partition.
Improved manageability: Lazy evaluation helps you organise your code into smaller logical steps while letting Spark decide the most efficient execution plan.
Optimisation opportunities: Spark’s Catalyst optimiser can reorder, merge or eliminate transformations before they run.

Example: Watching Lazy Evaluation in Action

Let’s see lazy evaluation in practice. In this example we load a CSV, apply several transformations and call an action at the end. Note that nothing happens until show() is executed:

```

from pyspark.sql import SparkSession

# Start a SparkSession

def create_spark_session():

return SparkSession.builder.appName("LazyEvalDemo").getOrCreate()

spark = create_spark_session()

# Load a CSV file (replace with your path)

df = spark.read.csv("s3a://pyspark420/uploads/b0cbebc8-4e6f-4544-95be-d7702f864d14.csv", header=True, inferSchema=True)

# Transformations: build the plan

high_salary = df.filter(df.salary > 60000) # filter rows

selected = high_salary.select("name", "salary") # select columns

sorted_df = selected.orderBy("salary", ascending=False) # sort

# At this point, Spark hasn't read the file or filtered anything!

# Action: triggers execution

sorted_df.show(3)

```

# Now Spark will read, filter, select and sort the data, then display the top 3 records.

The call to show(3) starts the computation. If we replace show(3) with first(), Spark may only read enough partitions to return the first row, leaving the rest of the dataset untouched.

Viewing the DAG and Execution Plan

You can inspect the DAG and understand how Spark plans to run your code. Use the explain() method on a DataFrame to see the logical and physical plans,) to view a graphical DAG:

# Explain the plan

1sorted_df.explain("formatted")

▶ Output will appear here.

# This prints the logical and physical plan, showing how Spark will perform the filter, select and sort operations.

In the Spark UI, each operation appears as a node. Stages are separated by shuffle boundaries, and tasks within a stage can run in parallel.The following infographic summarises lazy evaluation: transformations are queued up and execution begins only when an action is called.

Working with Lazy Evaluation

Chain transformations: Combine multiple transformations before calling an action. This reduces the number of jobs and data scans.
Avoid collecting large datasets: Be cautious with collect() or toPandas(), which pull all data to the driver. Instead, use take(n), show() or limit() for quick checks.
Cache when necessary: If you plan to reuse the same DataFrame multiple times, call .cache() or .persist() after the transformations and before the first action. This prevents Spark from recomputing the entire DAG each time.
Use the right actions: Actions like first(), take(n), or count() are efficient because they return small results. Avoid collect() on huge datasets.

Lazy Evaluation in PySpark