PySpark dropDuplicates() and distinct(): deduplicate rows

PySpark `distinct()` removes fully duplicate rows; `dropDuplicates(subset)` removes rows that are duplicates on a chosen subset of columns, keeping the first occurrence. To keep the latest/best row per key, combine a window `row_number()` with a filter instead.

distinct vs dropDuplicates

distinct() dedupes on ALL columns; dropDuplicates(["user_id"]) dedupes on just the listed columns. The row kept is arbitrary unless you order first.

Keep the latest per key

For deterministic dedup (e.g. latest login), use ROW_NUMBER() OVER (PARTITION BY key ORDER BY ts DESC) and keep rn = 1 rather than dropDuplicates.

dropDuplicates vs distinct, and deduplicating on a key

distinct() removes rows that are identical across every column. dropDuplicates() does the same when called with no arguments, but its real value is the subset form: df.dropDuplicates(['customer_id']) keeps one row per customer even when the other columns differ. The catch is that which row survives is not defined — Spark keeps whichever it encounters first, and that can change between runs. Never rely on it when the surviving row matters.

Keeping the latest record per key, deterministically

When you need a specific survivor — usually the most recent version of a record — use a window function instead. Partition by the key, order by the timestamp descending, assign row_number(), then keep rows where the number equals 1. That expresses the intent exactly and gives the same answer every run, which matters for incremental loads and CDC pipelines. If ties are possible, add a tiebreaker column to the ordering so the result stays deterministic.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "a"), (1, "a"), (2, "b")], ["id", "val"])

df.distinct().show()                 # drop fully-identical rows
df.dropDuplicates(["id"]).show()     # one row per id

distinct() removes identical rows; dropDuplicates(["id"]) keeps one row per id.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between distinct() and dropDuplicates() in PySpark?

distinct() dedupes across all columns; dropDuplicates(subset) dedupes only on the given columns, keeping one row per unique combination.

How do I keep the latest row per key in PySpark?

Use a window: row_number() over (partitionBy key orderBy timestamp desc) and filter rn = 1 — dropDuplicates keeps an arbitrary row.

Practice challenges

• Deduplicate User Login Sessions
• Find Duplicate Emails

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs