Incremental Customer ETL Pipeline
PySpark · Intermediate · about 85 minutes · 5 required tasks
The scenario
You maintain the customer pipeline for a reporting team. An ingestion bug is duplicating customers and inflating revenue, the reports need readable region names instead of codes, and updates now arrive as a stream of changes that must be applied without keeping stale rows. You will fix the join, clean the ingested records, enrich customers with a small lookup table, and apply the latest change per customer. Each task is a self-contained practice dataset; together they follow the order the pipeline is built in.
What you will build
- A corrected orders-to-customers join and a de-duplicated customer table
- Customers enriched with region names using a broadcast join
- A latest-record-per-customer view and an applied CDC update set
Skills practiced
- DataFrame joins
- dropDuplicates
- Window functions (row_number)
- Broadcast joins
- CDC-style incremental updates
Stages
Fix ingestion
Stop duplicate customer rows from inflating results and clean the ingested records.
2 required tasks
Enrich with reference data
Add region names to customers with a broadcast join against a small lookup table.
1 required task, 1 optional stretch task
Apply incremental updates
Keep the latest record per customer and apply CDC updates to the main table.
2 required tasks, 1 optional stretch task
Every task opens in the regular auto-graded challenge editor. Tasks you have already solved count automatically once you sign in.