Partition Basics

Spark tutorial · PySpark.in

How Spark Stores DataFrames Internally

Unlike Pandas, Spark does not keep a DataFrame as one giant block in memory. Instead it breaks the data into smaller chunks called partitions. These partitions are spread across the cluster so multiple tasks can process them in parallel. According to the RDD programming guide, an RDD (Resilient Distributed Dataset) is a collection of elements partitioned across the nodes of the cluster, enabling parallel operations. Spark DataFrames sit on top of RDDs, so they inherit this partitioned structure.

What is Partitions in Spark?

A partition is a subset of your DataFrame’s rows stored on one node of the cluster. You can think of a partition like a chapter in a book:

• Each chapter is a partition.
• All chapters together form the book (the DataFrame).

Because data is split into partitions, Spark can schedule one task per partition, processing multiple chunks at the same time. This improves execution speed, memory efficiency and fault tolerance. If one partition fails, Spark only needs to recompute that chunk instead of the entire dataset.

Why Partitions Are Important

How Spark Decides Number of Partitions

Spark uses configuration defaults to determine partition counts:

• Shuffle operations (joins, groupBy, aggregations) use spark.sql.shuffle.partitions, which defaults to 200 partitions.
• RDD operations use spark.default.parallelism. This defaults to the total number of CPU cores in the cluster (or your local machine).

You can override these values at runtime to suit your workload.

Check Current Number of Partitions

To see how many partitions a DataFrame has, use the underlying RDD’s getNumPartitions():

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("CheckPartitions") \

.master("local[*]") \

.getOrCreate()

data = [("Laptop", 55000), ("Mouse", 500), ("Keyboard", 1200)]

df = spark.createDataFrame(data, ["product", "price"])

print("Partitions:", df.rdd.getNumPartitions())

```

View Data Inside Each Partition

To view the rows in each partition, use qlom() to collect the data into lists:

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("CheckPartitions") \

.master("local[*]") \

.getOrCreate()

data = [("Laptop", 55000), ("Mouse", 500), ("Keyboard", 1200)]

df = spark.createDataFrame(data, ["product", "price"])

print(df.rdd.glom().collect())

```

Adjusting Partitions

Spark offers two methods to change the number of partitions:

repartition()

• Increases or decreases partitions by shuffling the data across the cluster.
• Use when you need to add parallelism or fix data skew before wide transformations (joins, aggregations).
• Because it triggers a shuffle, it’s more expensive than coalesce().

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("CheckPartitions") \

.master("local[*]") \

.getOrCreate()

data = [("Laptop", 55000), ("Mouse", 500), ("Keyboard", 1200)]

df = spark.createDataFrame(data, ["product", "price"])

print(df.rdd.glom().collect())

```

Coalesce

• Reduces the number of partitions without a full shuffle; it simply merges existing partitions.
• Faster than  repartition() and ideal after filtering or when preparing data for writing.
• Cannot increase partitions.

```

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.appName("CheckPartitions") \

.master("local[*]") \

.getOrCreate()

data = [("Laptop", 55000), ("Mouse", 500), ("Keyboard", 1200)]

df = spark.createDataFrame(data, ["product", "price"])

df3 = df.coalesce(1)

print(df3.rdd.getNumPartitions())

```

When to Use Each

Defaults and Best Practices

• Default shuffle partitions: spark.sql.shuffle.partitions = 200. Increase for large clusters or decrease for small jobs to avoid too many tiny tasks.
• Default parallelism for RDDs: Number of cores in your cluster. Use this to control parallelize() and RDD operations.
• Choose repartition() when you need more parallelism or to redistribute skewed data before a wide transformation.
• Choose coalesce() when you only need to reduce partitions and avoid shuffling.ks

More Spark tutorials

• about pyspark
• text diagram
• Apache Spark Runtime Architecture
• Introduction to RDD
• Actions vs Transformations
• Lazy Evaluation in PySpark

All tutorials · Try the free PySpark compiler · Practice challenges