Understanding Apache Spark RDDs

Spark tutorial · PySpark.in

Resilient Distributed Datasets (RDDs)

RDD as the foundation of Apache Spark.

RDD (Resilient Distributed Dataset) is Spark’s core distributed data structure that stores partitioned data across multiple nodes, supports parallel processing, lazy transformations, fault tolerance through lineage, and in-memory computation for fast large-scale data processing.

Imagine you have a huge file with millions of records. A single machine may struggle to process it quickly. Spark solves this by splitting the data into small chunks, distributing those chunks across multiple machines, and processing them in parallel.

That distributed collection of data is called an RDD.

What does RDD mean?

R — Resilient

Resilient means fault tolerant.

If one machine fails while processing data, Spark does not lose everything.

Spark can rebuild only the missing data using the history of how the RDD was created.

That history is called lineage.

D — Distributed

Distributed means the data is not stored on one machine.

It is divided into smaller pieces called partitions, and those partitions are spread across multiple machines.

Each machine works on its own partition.

D — Dataset

Dataset simply means a collection of data.

That data could be:

• numbers
• strings
• log records
• CSV rows
• JSON documents

Simple Real-Life Example

Suppose you have 100 GB of sales data.

Without Spark:

• One machine reads all 100 GB
• Processing is slow

With Spark:

• Spark splits data into smaller parts (called partitions)
• Different machines process different partitions at the same time

For example:

Now all machines process data at the same time.

Instead of one person reading 100 books, Spark uses 100 people reading 1 book each.

That is the power of RDD.

How to Create an RDD

There are two common ways.

1. From Existing Data in Python

You can create an RDD from a normal Python list.

If you already have Python data, you can convert it into an RDD.

```

from pyspark.sql import SparkSession

# Create Spark session

spark = SparkSession.builder \

.appName("SimpleRDDExample") \

.master("local[*]") \

.getOrCreate()

# Create SparkContext

sc = spark.sparkContext

# Sample data

data = [1, 2, 3, 4, 5]

# Create RDD

rdd = sc.parallelize(data)

print(rdd.collect())

# Stop Spark session

spark.stop()

```

What happens here?

• parallelize() takes the Python list
• Spark divides it into partitions
• Now the data can be processed in parallel

What happens internally?

Step 1

• Python list exists in driver memory.
• [1, 2, 3, 4, 5]

Step 2

• parallelize() converts it into a distributed dataset.
• Spark breaks it into partitions.
• Example:

• Now Spark can process them in parallel.

2. From External Files

Most real projects load data from files.

```

from pyspark.sql import SparkSession

# Create Spark session

spark = SparkSession.builder \

.appName("RDDFromFile") \

.master("local[*]") \

.getOrCreate()

sc = spark.sparkContext

# Read file

rdd = sc.textFile("s3a://pyspark420/pyspark-upload/employees.txt")

# Now each line in the file becomes one record in the RDD.

# Print raw file data

print("Raw file data:")

print(rdd.collect())

# Convert each line into columns

parsed_rdd = rdd.map(lambda line: line.split(","))

print("Parsed data:")

print(parsed_rdd.collect())

spark.stop()

```

Partitions — Very Important

Important point

Each line becomes one element of the RDD.

If the file contains:

101,John,50000
102,Alice,60000
103,Bob,55000

Then RDD looks like:

[
"101,John,50000",
"102,Alice,60000",
"103,Bob,55000"
]

Parsing File Data

Usually we split rows into columns.

parsed = rdd.map(lambda line: line.split(","))

print(parsed.collect())

Output:

[
['101', 'John', '50000'],
['102', 'Alice', '60000'],
['103', 'Bob', '55000']
]

If Spark creates 3 partitions, a possible layout is:

So internally:

rdd

can be visualized as:

Partition 1 → ["101,John,50000", "102,Alice,60000"]
Partition 2 → ["103,Bob,55000", "104,Anna,70000"]
Partition 3 → ["105,David,65000", "106,Sam,52000"]

After map()

When this transformation runs:

parsed_rdd = rdd.map(lambda line: line.split(","))

the number of partitions stays the same.

Only the contents inside each partition change.

Important point

map() does not create new partitions.

It processes records inside existing partitions.

So Spark sees it like this:

Read file → create partitions
Partition 1
Partition 2
Partition 3

Apply map() inside each partition

Parallel execution

Each partition becomes one task.

That means Apache Spark can process all three partitions at the same time.

Task 1 → Partition 1
Task 2 → Partition 2
Task 3 → Partition 3

How to check actual partitions

You can verify the partition count in your program:

```

print(rdd.getNumPartitions())

```

And you can inspect partition contents like this:

```

print(rdd.glom().collect())

```

Example output:

[
['101,John,50000', '102,Alice,60000'],
['103,Bob,55000', '104,Anna,70000'],
['105,David,65000', '106,Sam,52000']
]

This creates 2 partitions.

Why partitions matter

More partitions → more parallelism
Too many partitions → more overhead

A common rule:

2 to 4 partitions per CPU core

RDD Operations

RDD supports two kinds of operations.

1. Transformations

Transformations create a new RDD. They do not execute immediately.

Examples:

• map()
• filter()
• flatMap()
• distinct()

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("TransformationExample") \

.master("local[*]") \

.getOrCreate()

sc = spark.sparkContext

# Employee names

employees = sc.parallelize(["john", "alice", "bob", "anna"])

# Transformation 1: convert names to uppercase

upper_names = employees.map(lambda name: name.upper())

# Transformation 2: keep names starting with A

a_names = upper_names.filter(lambda name: name.startswith("A"))

# Spark still has not executed anything yet

print("Transformations created, but not executed.")

# Action triggers execution

print(a_names.collect())

spark.stop()

```

2. Actions

Actions trigger execution and return results.

Examples:

• collect()
• count()
• reduce()
• first()
• take()
• reduce()

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("RDDActionsExample") \

.master("local[*]") \

.getOrCreate()

sc = spark.sparkContext

# Create RDD

sales = sc.parallelize([1200, 1500, 1800, 2000, 1700])

# Transformation (not executed yet)

high_sales = sales.filter(lambda x: x > 1500)

# Actions trigger execution

print("All high sales:", high_sales.collect())

print("Number of high sales:", high_sales.count())

print("First high sale:", high_sales.first())

print("Total high sales:", high_sales.reduce(lambda a, b: a + b))

spark.stop()

```

Lazy Evaluation — Spark’s Superpower

One of the biggest reasons Spark is fast is lazy evaluation.

What it means

When you write a transformation like map(), filter(), or flatMap(), Spark does not run it immediately.

Instead, Spark only remembers what needs to be done.

It waits until you call an action such as:

• collect()
• count()
• reduce()
• take()

Only then does Spark actually execute the work.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("LazyEvaluationExample") \

.master("local[*]") \

.getOrCreate()

sc = spark.sparkContext

# Create RDD

numbers = sc.parallelize([1, 2, 3, 4, 5, 6])

# Transformations

doubled = numbers.map(lambda x: x * 2)

filtered = doubled.filter(lambda x: x > 5)

print("Transformations created.")

print("No computation has happened yet.")

# Action

result = filtered.collect()

print("Result:", result)

spark.stop()

```

Why lazy evaluation is useful

In normal programming, each line often runs immediately.

For example, imagine processing a file step by step:

Read file
Store result
Apply map()
Store result
Apply reduce()
Store result

That approach can be expensive because every step may create temporary data.

In Apache Spark, transformations like map() and filter() do not run immediately.

Spark waits until an action such as collect(), count(), or reduce() is called.

Why this helps

Because Spark can look at the entire chain of operations first and build the most efficient execution plan.

Instead of executing each line one by one, Spark can combine steps.

This reduces

• Memory usage — fewer temporary intermediate objects
• Disk I/O — less writing/reading intermediate results
• Execution time — fewer separate passes over the data

Example

```

hide

numbers = sc.parallelize([1, 2, 3, 4, 5])

result = numbers.map(lambda x: x * 2) \
.filter(lambda x: x > 5) \
.collect()

```

What happens?

When Spark sees:

numbers.map(...)
filter(...)

it does not run those steps yet.

It simply remembers:

RDD → map → filter → collect

Only when collect() runs does Spark start processing.

Partition-level view

Suppose the RDD has 2 partitions.

Step 1 — map(lambda x: x * 2)

Step 2 — filter(lambda x: x > 5)

Final collect()

[6, 8, 10]

Spark processes each partition in parallel.

Intermediate view — what Spark sees

Spark does not think like this:

Run line 1
Run line 2
Run line 3

Instead, it builds a logical plan.

For the example above, Spark sees:

numbers → map → filter → collect

That entire chain is analyzed before execution.

Advanced view — DAG

Internally, Spark creates a DAG.

Directed Acyclic Graph means a graph that shows dependencies between operations.

For example:

textFile → map → filter → reduce

Why DAG matters

Spark uses the DAG to decide:

1. Task scheduling

Which partition should run on which executor.

2. Stage boundaries

Some operations can be grouped together.

For example:

textFile → map → filter

may run in one stage.

But operations like reduceByKey() often require shuffle, which creates a new stage.

3. Optimization opportunities

Spark can combine narrow transformations into a single pipeline.

That is one major reason Apache Spark is fast.

Why RDDs are fault tolerant

Suppose an RDD has 4 partitions.

Now imagine Machine 3 crashes.

Partition 3 is lost.

What does Spark do?

Spark does not reload everything.

Instead, Spark uses lineage.

What is lineage?

Lineage means Spark remembers:

• where the data came from
• what transformations created each partition

Example:

employees.txt → map → filter → reduce

So if Partition 3 is lost, Spark only recomputes Partition 3.

It does not recompute Partition 1, 2, and 4.

That is why RDD means Resilient Distributed Dataset.

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges