Remove Duplicate Customer Records (PySpark)

PYSPARK coding challenge · Difficulty: medium · Topic: Deduplication with ROW_NUMBER · +110 XP

Problem

A data-ingestion fault inserted some customers more than once.

Two rows are duplicates only when all three of these are identical:

customer_name, email, phone.

Keep exactly one row per customer: the one with the smallest customer_id.

Schema — `customers`

| Column | Type |
| --- | --- |
| customer_id | int |
| customer_name | string |
| email | string |
| phone | string |

Example Input — `customers`

| customer_id | customer_name | email | phone |
| --- | --- | --- | --- |
| 101 | Alice | alice@mail.com | 555-3311 |
| 102 | Bob | bob@mail.com | 555-3322 |
| 103 | Alice | alice@mail.com | 555-3311 |
| 104 | Carol | carol@mail.com | 555-3333 |
| 105 | Bob | bob@mail.com | 555-3322 |
| 106 | Alice | alice@mail.com | 555-3311 |
| 107 | Dan | dan@mail.com | 555-3344 |
| 108 | Dan | dan@mail.com | 555-3399 |

Expected Output

| customer_id | customer_name | email | phone |
| --- | --- | --- | --- |
| 101 | Alice | alice@mail.com | 555-3311 |
| 102 | Bob | bob@mail.com | 555-3322 |
| 104 | Carol | carol@mail.com | 555-3333 |
| 107 | Dan | dan@mail.com | 555-3344 |
| 108 | Dan | dan@mail.com | 555-3399 |

Explanation

Alice appears three times — ids 101, 103 and 106 — with the same name, email and

phone. Keep id 101, the smallest. Drop 103 and 106.

Bob appears twice — ids 102 and 105. Keep id 102.

Carol appears once — keep it.

Dan is the interesting one. Ids 107 and 108 share a name *and* an email, but the

phones differ: 555-3344 against 555-3399. All three columns must match for a

row to be a duplicate, so these are two different records and both are kept.

5 rows survive out of 8.

A note on `dropDuplicates`

customers.dropDuplicates(["customer_name", "email", "phone"]) keeps an

arbitrary row from each group. On a small local DataFrame it often happens

to keep the first one, but that is not guaranteed once the data is partitioned

and shuffled, so it is not a safe way to say "keep the smallest id".

row_number() ordered by customer_id makes the choice explicit.

Notes

What this PYSPARK challenge teaches you

“Remove Duplicate Customer Records (PySpark)” is a medium-level PYSPARK challenge focused on Deduplication with ROW_NUMBER. Working through it gives you hands-on practice with row_number, Window, Deduplication, partitionBy, Data Cleaning — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

  1. dropDuplicates keeps an arbitrary row per group, so it cannot promise the smallest customer_id.
  2. Window.partitionBy("customer_name","email","phone").orderBy("customer_id"), then F.row_number().over(w) == 1, then drop the helper column.

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated medium and covers Deduplication with ROW_NUMBER.

Solve this challenge free on PySpark.in