PySpark dropDuplicates() and distinct(): deduplicate rows

PySpark `distinct()` removes fully duplicate rows; `dropDuplicates(subset)` removes rows that are duplicates on a chosen subset of columns, keeping the first occurrence. To keep the latest/best row per key, combine a window `row_number()` with a filter instead.

distinct vs dropDuplicates

distinct() dedupes on ALL columns; dropDuplicates(["user_id"]) dedupes on just the listed columns. The row kept is arbitrary unless you order first.

Keep the latest per key

For deterministic dedup (e.g. latest login), use ROW_NUMBER() OVER (PARTITION BY key ORDER BY ts DESC) and keep rn = 1 rather than dropDuplicates.

Example (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, "a"), (1, "a"), (2, "b")], ["id", "val"])

df.distinct().show()                 # drop fully-identical rows
df.dropDuplicates(["id"]).show()     # one row per id

distinct() removes identical rows; dropDuplicates(["id"]) keeps one row per id.

Run this example in the free online PySpark compiler

Frequently asked questions

What is the difference between distinct() and dropDuplicates() in PySpark?

distinct() dedupes across all columns; dropDuplicates(subset) dedupes only on the given columns, keeping one row per unique combination.

How do I keep the latest row per key in PySpark?

Use a window: row_number() over (partitionBy key orderBy timestamp desc) and filter rn = 1 — dropDuplicates keeps an arbitrary row.

Practice challenges

• Deduplicate User Login Sessions
• Find Duplicate Emails

Open the free PySpark compiler · Data Engineering challenges · Data Engineering jobs