Implement SCD Type 2 for Customer Data with Delta Lake

PYSPARK coding challenge · Difficulty: easy · Topic: Delta Lake MERGE / SCD Type 2 · +50 XP

Problem

You have a customer table where customer details like name and address can change over time. Implement Slowly Changing Dimension (SCD) Type 2 to track historical data. Use Delta Lake's MERGE functionality to update the table such that the latest entry for a customer is marked as 'current'.

Example Input - `customer_data`

| customer_id | name | address | valid_from | valid_to | is_current |
| --- | --- | --- | --- | --- | --- |
| 1 | Alice | 123 Main St | 2023-01-01 | null | True |
| 1 | Alice | 456 Elm St | 2023-03-01 | 2023-06-01 | False |
| 2 | Bob | 789 Market St | 2023-02-01 | null | True |
| 2 | Bob | 789 Market St | 2023-02-01 | 2023-05-01 | False |
| 3 | Charlie | 1010 Broadway | 2023-03-01 | null | True |
| 3 | Charlie | 2020 Oak St | 2023-04-01 | 2023-07-01 | False |

Expected Output

| customer_id | name | address | is_current |
| --- | --- | --- | --- |
| 1 | Alice | 123 Main St | true |
| 2 | Bob | 789 Market St | true |
| 3 | Charlie | 1010 Broadway | true |

Notes

• The DataFrame is created for you - do not recreate it
• Build a DataFrame called df_result and finish with df_result.show()

What this PYSPARK challenge teaches you

“Implement SCD Type 2 for Customer Data with Delta Lake” is a easy-level PYSPARK challenge focused on Delta Lake MERGE / SCD Type 2. Working through it gives you hands-on practice with Delta Lake, MERGE, SCD Type 2 — the kind of transformation you are asked to write in real data engineering work and in technical interviews. You can solve it directly in the browser: the dataset is pre-loaded, so you write the query or DataFrame code, run it, and compare your output against the expected result immediately.

Concepts covered

• Delta Lake
• MERGE
• SCD Type 2

How to approach it

If you get stuck, work through these steps in order before looking at a full solution — each one narrows the problem down:

1. 1 ≤ number of rows ≤ 1,000,000
2. Each customer_id appears between 1 and 10 times
3. valid_from and valid_to are strings in 'YYYY-MM-DD' format
4. is_current is a boolean indicating if the record is the latest for that customer
5. Use Delta Lake for merge operations

How to practise it on PySpark.in

Open the challenge, write your PySpark code in the editor and press Run to execute it against the sample dataset. Submitting checks your output against every test case, including hidden ones, so you find out straight away whether your logic holds up. You can retry as often as you like, and each solved challenge adds to your XP.

Related PYSPARK challenges

• Fix the Broken Pipeline
• Optimize the 100M-Row Join
• Parse Apache Logs with Regex
• Optimize Small DataFrame Join with Broadcast
• Optimize Average Rating Calculation for Products
• Deduplicate and Aggregate User Actions with Latest Session

Frequently asked questions

Do I need to install Spark or a database to solve this?

No. The PYSPARK environment runs in your browser with the sample data already loaded, so there is nothing to install or configure.

Is this challenge free?

Yes - the problem, the sample dataset, the hints and unlimited test runs are free.

What level is it?

It is rated easy and covers Delta Lake MERGE / SCD Type 2.

Solve this challenge free on PySpark.in