Incremental Customer ETL
PySpark · Intermediate · about 85 minutes · 5 required tasks
The scenario
You maintain the customer pipeline for a reporting team. An ingestion bug is duplicating customers and inflating revenue, the reports need readable region names instead of codes, and updates now arrive as a stream of changes that must be applied without keeping stale rows. You will fix the join, clean the ingested records, enrich customers with a small lookup table, and apply the latest change per customer. Each task is a self-contained practice dataset; together they follow the order the pipeline is built in.
What you will build
- A corrected orders-to-customers join and a de-duplicated customer table
- Customers enriched with region names using a broadcast join
- A latest-record-per-customer view and an applied CDC update set
Skills practiced
- DataFrame joins
- dropDuplicates
- Window functions (row_number)
- Broadcast joins
- CDC-style incremental updates
Before you start
- PySpark DataFrame basics: select, filter and withColumn
- DataFrame joins and join keys
- Window functions with row_number()
The data
A small PySpark practice DataFrame per task: customers and orders, a region lookup table, and a set of change records with update timestamps.
Stages
Fix ingestion
Stop duplicate customer rows from inflating results and clean the ingested records.
2 required tasks
Enrich with reference data
Add region names to customers with a broadcast join against a small lookup table.
1 required task, 1 optional stretch task
Apply incremental updates
Keep the latest record per customer and apply CDC updates to the main table.
2 required tasks, 1 optional stretch task
Every task opens in the regular auto-graded challenge editor. Tasks you have already solved count automatically once you sign in.