Introduction to RDD

Spark tutorial · PySpark.in

Introduction to RDD in Apache Spark

Before Spark, Hadoop MapReduce dominated big data processing. However, MapReduce had some major limitations:

Slow processing: Every operation wrote to disk (HDFS), making iterative jobs slow.
Limited APIs: Only map and reduce operations; complex workflows were difficult.
No in-memory computation: Re-computation always started from scratch if data was reused.

Spark introduced RDDs to overcome these issues:

In-memory processing → much faster than disk I/O.
Lineage-based fault tolerance → lost data can be recomputed.
Rich transformations → map, filter, reduceByKey, join, etc.
Ideal for iterative algorithms → ML and graph processing become faster.

What is an RDD?

RDD = Resilient Distributed Dataset

Dataset: Collection of records (like Python lists or Java arrays).
Distributed: Data is split into partitions across nodes for parallel processing.
Resilient (Fault-Tolerant): If a partition is lost, Spark recomputes it using lineage.
Type-Safe: Define RDD[Int], RDD[String], etc., to catch errors early.
Supports Structured & Unstructured Data:
- Unstructured → logs, text files, tweets
- Structured → tables with rows & columns
Lazy Evaluation: Transformations (map, filter) don’t run until an action (count(), collect()) is triggered.

Analogy: RDD as a Classroom of Students

Imagine you are a teacher with 1000 students:

Checking marks alone → takes too long.
Divide students into 10 groups → assign 10 helpers.
Each helper checks their group in parallel.
Results are combined and returned to you.

RDD works the same way:

Splits big data into small partitions
Processes partitions in parallel
Combines results at the end

Tip: Use RDDs for low-level control or unstructured data. For most modern workloads, prefer DataFrames/Datasets.

When to Use RDDs

Low-level transformations not supported by DataFrames
Unstructured data processing (logs, tweets, raw text)
Custom transformations requiring full control

Example: Word Count Using RDD:

```
from pyspark.sql import SparkSession
# Initialize SparkSession
spark = SparkSession.builder.appName("WordCountExample").getOrCreate()
# Hardcoded list of words (simulating text input)
data = [
"hello world this is pyspark",
"pyspark is great for big data processing",
"hello spark hello python",
"data engineering with pyspark and spark"
]
# Create RDD from hardcoded data
text_rdd = spark.sparkContext.parallelize(data)
# Split lines into words
words = text_rdd.flatMap(lambda line: line.split(" "))
# Filter for words containing 'spark'
spark_words = words.filter(lambda w: "spark" in w.lower())
# Map each word to (word, 1) and reduce counts
counts = spark_words.map(lambda x: (x, 1)).reduceByKey(lambda a, b: a + b)
# Action: get top results
print(counts.collect())
# Stop SparkSession
spark.stop()

```

Load Big Data → Spark reads large dataset (text_rdd = spark.sparkContext.textFile("data.txt"))
Split into Partitions → Data is divided across multiple nodes
Transformations (Lazy) → Operations like map(), filter() applied per partition
Actions Trigger Execution → collect(), count(), take() run transformations in parallel
Fault Tolerance → Lost partitions are recomputed using lineage

Introduction to RDD

Limitations of RDDs

Spark cannot optimize inside custom functions (like lambda).
Inefficient operations if not careful (e.g., filtering after shuffle).
Debugging large distributed RDD jobs is harder.

Modern Spark tip: Prefer DataFrames/Datasets, use RDDs for low-level or unstructured data.

Note:

RDD = Foundation of Spark → distributed, fault-tolerant, flexible
Ideal for low-level control and unstructured data
DataFrames/Datasets → optimized, easier, preferred for most tasks
Knowing RDDs helps understand how Spark works internally

Introduction to RDD