repartition() vs coalesce() in PySpark

Both `repartition()` and `coalesce()` change the number of partitions of a PySpark DataFrame. `repartition(n)` does a full shuffle and can increase or decrease partitions with even distribution; `coalesce(n)` only *reduces* partitions and avoids a full shuffle by merging existing ones — faster but can leave uneven partitions.

When to use each

Use coalesce() to cut the number of output files after a filter (cheap, no shuffle). Use repartition() when you need more partitions, even distribution, or to repartition by a column before a join/write.

repartition by column

df.repartition(8, "customer_id") shuffles rows so all rows for a key land in the same partition — useful before key-based joins or partitioned writes.

The fundamental difference: shuffle or no shuffle

repartition(n) performs a full shuffle and redistributes rows evenly across exactly n partitions. It can increase or decrease the partition count and it can repartition by column, repartition('country'), which co-locates matching rows. coalesce(n) only reduces the partition count and does it by merging existing partitions on the same executor, with no shuffle. That makes coalesce much cheaper, but the result can be unbalanced because merged partitions inherit whatever sizes they had.

Which to reach for in practice

Use coalesce when you are shrinking the output at the end of a job — for example collapsing a thousand small task outputs into a handful of files before writing. Use repartition when you need balanced partitions, when you are increasing parallelism before an expensive stage, or when data is skewed and one task is holding everything up. The classic mistake is coalesce(1) on a large dataset to get a single output file: because there is no shuffle, that forces the entire computation upstream to collapse to one task, and the job either crawls or runs out of memory. repartition(1) is slower to start but far safer, and writing several files is usually better still.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.range(0, 1000)

print(df.rdd.getNumPartitions())
print(df.coalesce(2).rdd.getNumPartitions())     # reduce, no shuffle
print(df.repartition(8).rdd.getNumPartitions())  # full shuffle, even

coalesce reduces partitions without a shuffle; repartition reshuffles into evenly-sized partitions.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between repartition and coalesce in PySpark?

repartition does a full shuffle and can increase or decrease partitions evenly; coalesce only reduces partitions and avoids a full shuffle by merging existing ones.

Is coalesce faster than repartition?

Usually yes when reducing partitions, because coalesce avoids a full shuffle — but it can produce uneven partitions. repartition is better when you need balance or more partitions.

How do I reduce the number of output files in Spark?

Call coalesce(n) before writing, e.g. df.coalesce(1).write... to merge into fewer files without a costly shuffle.

Practice challenges

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs