Implement SCD Type 2 for Customer Data with Delta Lake

PYSPARK coding challenge · Difficulty: easy · +50 XP

You have a customer table where customer details like name and address can change over time. Implement Slowly Changing Dimension (SCD) Type 2 to track historical data. Use Delta Lake's MERGE functionality to update the table such that the latest entry for a customer is marked as 'current'.

Example input

`

customer_data DataFrame:

customer_id | name | address | valid_from | valid_to | is_current

1 | Alice | 123 Main St | 2023-01-01 | NULL | True

1 | Alice | 456 Elm St | 2023-03-01 | 2023-06-01| False

2 | Bob | 789 Market St | 2023-02-01 | NULL | True

2 | Bob | 789 Market St | 2023-02-01 | 2023-05-01| False

3 | Charlie | 1010 Broadway | 2023-03-01 | NULL | True

3 | Charlie | 2020 Oak St | 2023-04-01 | 2023-07-01| False

`

Expected output

`

+-----------+--------+-------------+-----------+------------+-----------+

|customer_id|name    |address      |valid_from |valid_to    |is_current |

+-----------+--------+-------------+-----------+------------+-----------+

|          1|Alice   |123 Main St  |2023-01-01 |NULL        |True       |
|          2|Bob     |789 Market St|2023-02-01 |NULL        |True       |
|          3|Charlie |1010 Broadway|2023-03-01 |NULL        |True       |

+-----------+--------+-------------+-----------+------------+-----------+

`