repartition() vs coalesce() in PySpark
Both repartition() and coalesce() change the number of partitions of a PySpark DataFrame. repartition(n) does a full shuffle and can increase or decrease partitions with even distribution; coalesce(n) only *reduces* partitions and avoids a full shuffle by merging existing ones — faster but can leave uneven partitions.
When to use each
Use coalesce() to cut the number of output files after a filter (cheap, no shuffle). Use repartition() when you need more partitions, even distribution, or to repartition by a column before a join/write.
repartition by column
df.repartition(8, "customer_id") shuffles rows so all rows for a key land in the same partition — useful before key-based joins or partitioned writes.
The fundamental difference: shuffle or no shuffle
repartition(n) performs a full shuffle and redistributes rows evenly across exactly n partitions. It can increase or decrease the partition count and it can repartition by column, repartition('country'), which co-locates matching rows. coalesce(n) only reduces the partition count and does it by merging existing partitions on the same executor, with no shuffle. That makes coalesce much cheaper, but the result can be unbalanced because merged partitions inherit whatever sizes they had.
Which to reach for in practice
Reducing partitions can reduce small output files, but an aggressive coalesce(1) also limits parallelism. Repartitioning by a skewed key does not automatically balance the data: a popular key can still dominate a partition. Inspect partition sizes and the physical plan before choosing a count. Output-file counts can also depend on writer options and partitioned output.
| Question | repartition | coalesce |
|---|---|---|
| Increase partitions? | Yes | No |
| Reduce partitions? | Yes | Yes |
| Shuffle introduced? | Yes | No shuffle by coalesce itself |
| Distribution | By round-robin or supplied expressions | Merges existing partitions |
| Main tradeoff | Redistribution costs network work | Large reductions can reduce parallelism |
Example (PySpark)
from pyspark.sql import SparkSession
from pyspark.sql.functions import spark_partition_id
spark = SparkSession.builder.getOrCreate()
data = spark.range(0, 12, 1, 4)
for label, frame in [("original", data), ("coalesced", data.coalesce(2)), ("repartitioned", data.repartition(3))]:
print(label, "rows", frame.count())
frame.groupBy(spark_partition_id().alias("partition")).count().orderBy("partition").show()
Runs only 12 rows. Every operation retains 12 rows; inspect the per-partition counts printed after each label. Distribution can depend on the plan, so the table is an observation rather than a performance benchmark.
Run this example in the free online PySpark compiler
Expected output
original rows 12
[per-partition counts]
coalesced rows 12
[per-partition counts]
repartitioned rows 12
[per-partition counts]Frequently asked questions
What is the difference between repartition and coalesce in PySpark?
repartition does a full shuffle and can increase or decrease partitions evenly; coalesce only reduces partitions and avoids a full shuffle by merging existing ones.
Is coalesce faster than repartition?
Usually yes when reducing partitions, because coalesce avoids a full shuffle — but it can produce uneven partitions. repartition is better when you need balance or more partitions.
How do I reduce the number of output files in Spark?
Call coalesce(n) before writing, e.g. df.coalesce(1).write... to merge into fewer files without a costly shuffle.
Practice challenges
Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs