PySpark dropDuplicates() and distinct(): deduplicate rows

PySpark distinct() removes fully duplicate rows; dropDuplicates(subset) removes rows that are duplicates on a chosen subset of columns, keeping the first occurrence. To keep the latest/best row per key, combine a window row_number() with a filter instead.

distinct vs dropDuplicates

df.distinct() removes fully identical rows. df.dropDuplicates(["id"]) compares only the listed columns. It does not promise which values survive in other columns; sorting before deduplication is not a reliable survivor-selection rule.

Keep the latest per key

For deterministic dedup (e.g. latest login), use ROW_NUMBER() OVER (PARTITION BY key ORDER BY ts DESC) and keep rn = 1 rather than dropDuplicates.

dropDuplicates vs distinct, and deduplicating on a key

distinct() and dropDuplicates() without a subset compare every column. A subset changes the definition of a duplicate: df.dropDuplicates(["customer_id"]) retains one row per customer key even when other fields differ. If those fields matter, explicitly select a survivor with a window rather than relying on input order.

Keeping the latest record per key, deterministically

When you need a specific survivor — usually the most recent version of a record — use a window function instead. Partition by the key, order by the timestamp descending, assign row_number(), then keep rows where the number equals 1. That expresses the intent exactly and gives the same answer every run, which matters for incremental loads and CDC pipelines. If ties are possible, add a tiebreaker column to the ordering so the result stays deterministic.

Choose a deduplication rule
Goal	Operation	Surviving values
Remove identical rows	distinct() or dropDuplicates()	All columns match
One row per key	dropDuplicates(["id"])	Other column values are unspecified
Latest row per key	row_number() over an ordered window	Use a stable tie-breaker when timestamps tie

Example (PySpark)

from pyspark.sql import SparkSession, Window
from pyspark.sql import functions as F

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([
    (1, "old", "2026-01-01", 10),
    (1, "new", "2026-01-02", 11),
    (1, "tie", "2026-01-02", 12),
    (2, "only", "2026-01-01", 20),
    (2, "only", "2026-01-01", 20),
], ["id", "value", "updated_at", "event_id"])

print("unique rows", df.distinct().count())
print("unique keys", df.dropDuplicates(["id"]).count())
order = Window.partitionBy("id").orderBy(
    F.col("updated_at").desc(), F.col("event_id").desc()
)
latest = df.withColumn("rn", F.row_number().over(order)).filter("rn = 1")
for row in latest.orderBy("id").collect():
    print(row.id, row.value, row.event_id)

Five input rows become four distinct rows or two distinct keys. The ordered window selects event 12 for id 1 when the dates tie. Fixed-width ISO date strings sort correctly in this small example; parse real timestamps to the appropriate timestamp type. A tie-breaker must fully determine the surviving values you care about.

Run this example in the free online PySpark compiler

Expected output

unique rows 4
unique keys 2
1 tie 12
2 only 20

Frequently asked questions

What is the difference between distinct() and dropDuplicates() in PySpark?

distinct() dedupes across all columns; dropDuplicates(subset) dedupes only on the given columns, keeping one row per unique combination.

How do I keep the latest row per key in PySpark?

Use a window: row_number() over (partitionBy key orderBy timestamp desc) and filter rn = 1 — dropDuplicates keeps an arbitrary row.

Practice challenges

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs