Data Engineering Foundations

A beginner-friendly path mixing SQL and PySpark: the same data-engineering fundamentals -- filtering, aggregation, joins, deduplication, window functions and incremental processing -- practiced in both languages side by side.

Sign in to start this path and track your progress.

Stages

1. 1Data Processing Basics Not StartedFiltering rows in SQL and PySpark, side by side.▲
   →Find All Products That Are Out of StockEasy○Filter Numeric Values Stored as Strings (SQL)Easy○Select Two Columns from a DataFrameEasy○Filter Rows with a ConditionEasy

   Optional Practice

   ○Add a Bonus Column with withColumnEasy○Sort by Salary DescendingEasy
2. 2Filtering & Aggregation Not StartedGROUP BY / groupBy aggregation in both languages.▼
3. 3Joins Not StartedCombining related tables/DataFrames in SQL and PySpark.▼
4. 4Data Cleaning & Deduplication Not StartedRemoving duplicate records and cleaning raw text.▼
5. 5Window Functions for Pipelines Not StartedRunning totals and day-over-day change -- the window-function patterns that show up constantly in real pipelines.▼
6. 6Incremental Processing & CDC / SCD Not StartedSlowly Changing Dimensions and incremental/CDC loads. The catalog currently has few published challenges here -- reported as a content gap, not forced.▼
7. 7Data Quality & Pipeline Thinking Not StartedPivoting data for reporting and computing product-level activity metrics. Also a thin stage in the current catalog -- reported as a content gap.▼
8. 8Real-World Data Engineering Scenarios Not StartedApplied, larger-scale problems: join performance at scale, and retention/churn computed the same way in both languages.▼