repartition() vs coalesce() in PySpark

Both `repartition()` and `coalesce()` change the number of partitions of a PySpark DataFrame. `repartition(n)` does a full shuffle and can increase or decrease partitions with even distribution; `coalesce(n)` only *reduces* partitions and avoids a full shuffle by merging existing ones — faster but can leave uneven partitions.

When to use each

Use coalesce() to cut the number of output files after a filter (cheap, no shuffle). Use repartition() when you need more partitions, even distribution, or to repartition by a column before a join/write.

repartition by column

df.repartition(8, "customer_id") shuffles rows so all rows for a key land in the same partition — useful before key-based joins or partitioned writes.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.range(0, 1000)

print(df.rdd.getNumPartitions())
print(df.coalesce(2).rdd.getNumPartitions())     # reduce, no shuffle
print(df.repartition(8).rdd.getNumPartitions())  # full shuffle, even

coalesce reduces partitions without a shuffle; repartition reshuffles into evenly-sized partitions.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between repartition and coalesce in PySpark?

repartition does a full shuffle and can increase or decrease partitions evenly; coalesce only reduces partitions and avoids a full shuffle by merging existing ones.

Is coalesce faster than repartition?

Usually yes when reducing partitions, because coalesce avoids a full shuffle — but it can produce uneven partitions. repartition is better when you need balance or more partitions.

How do I reduce the number of output files in Spark?

Call coalesce(n) before writing, e.g. df.coalesce(1).write... to merge into fewer files without a costly shuffle.

Practice challenges

• Optimize the 100M-Row Join

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs